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IMPROVED PICTURE/GRAPHICS CLASSIFICATION 
SYSTEM AND METHOD 

Background of the Invention 

[0001] The present invention relates to image processing. It 

finds particular application in conjunction with classification of images between natural 
pictures and synthetic graphics, and will be described with particular reference thereto. 
However, it is to be appreciated that the present invention is also amenable to other 
like applications. 

[0002] During the past several decades, products and services 

such as TVs, video monitors, photography, motion pictures, copying devices, 
magazines, brochures, newspapers, etc. have steadily evolved from monochrome to 
color. With the increasing use of color products and services, there is a growing 
demand for "brighter" and more "colorful" colors in several applications. Due to this 
growing demand, display and printing of color imagery that is visually pleasing has 
become a very important topic. In a typical color copier application, the goal is to 
render the scanned document in such a way that it is most pleasing to the user. 

[0003] Natural pictures differ from synthetic graphics in many 

aspects, both in terms of visual perception and image statistics. Synthetic graphics are 
featured with smooth regions separated by sharp edges. On the contrary, natural 
pictures are often noisier and the region boundaries are less prominent. In processing 
scanned images, it is sometime beneficial to distinguish images from different origins 
(e.g., synthetic graphics or natural pictures), however, the origin or "type" information 
about a scanned image is usually unavailable. The "type" information should be 
automatically extracted from the scanned image. This "type" information is then used 
in further processing of the images. High-level image classification can be achieved by 
analysis of low-level image attributes geared for the particular classes. Coloring 
schemes (e.g., gamut-mapping or filtering algorithms) are tailored for specific types of 
images to obtain quality reproduction. Once an image has been identified as a graphics 



image, farther identification of image characteristics can be used to fine-tune the 
coloring schemes for more appealing reproductions. The most prominent 
characteristics of a graphics image include patches or areas of the image with uniform 
color and areas with uniformly changing colors. These areas of uniformly changing 
color are called sweeps. 

[0004] Picture/graphics classifiers have been developed to 

differentiate between a picture image and a graphics image by analyzing low-level image 
statistics. For example, U.S. Patent No. 5,767,978 to Revankar et al. discloses an 
adaptable image segmentation system for differentially rendering black and white and/or 
color images using a plurality of imaging techniques. An image is segmented according 
to classes of regions that may be rendered according to the same imaging techniques. 
Image regions may be rendered according to a three-class system (such as traditional 
text, graphic, and picture systems), or according to more than three (3) image classes. 
In addition, only two (2) image classes may be required to render high quality draft or 
final output images. The image characteristics that may be rendered differently from 
class to class may include half toning, colorization and other image attributes. 

[0005] Graphics are typically generated using a limited number 

of colors, usually containing only a few areas of uniform colors. On the other hand, 
natural pictures are more noisy, containing smoothly varying colors. A picture/graphics 
classifier can analyze the colors to distinguish between picture and graphics images. 

[0006] Graphics images contain several areas of uniform color, 

lines drawings, text, and have very sharp, prominent, long edges. On the other hand, 
natural pictures are very noisy and contain short broken edges, A picture/graphics 
classifier can analyze statistics based on edges to distinguish between picture and 
graphics images. 

[0007] Classifiers that can be used to solve a certain 

classification problem include statistical, structural, neural networks, fuzzy logic, and 
machine learning classifiers. Several of these classifiers are available in public domain 
and commercial packages. However, no single classifier seems to be highly successful 
in dealing with complex real world problems. Each classifier has its own weaknesses 
and strengths. 



[0008] The picture/graphics classification methods described 

above each use features of the image to make a "binary" classification decision (i.e., 
picture or graphics). The binary classification result is then used to "switch" between 
image processing functions. However, using the current set of features and the binary 
classification scheme, the classification accuracy, as tested on large image sets, is not 
perfect. Even with improved features and the binary classification scheme, it may not 
be possible to achieve perfect classification. In fact, there are images for which a clear 
classification cannot even be made by a human observer. Under such circumstances, 
the binary decision is often wrong, and could lead to objectionable image artifacts. 

[0009] U.S. Patent No. 5,778,156 to Schweid et al. discloses an 

improved method of image processing utilizing a fuzzy logic classification process. The 
disclosure includes a system and method to electronically image process a pixel 
belonging to a set of digital image data with respect to a membership of the pixel in a 
plurality of image classes. This process uses classification to determine a membership 
value for the pixel for each image class and generates an effect tag for the pixel based 
on the fuzzy classification determination. The pixel is image processed based on the 
membership vector of the pixel. The image processing may include screening and 
filtering. The screening process screens the pixel by generating a screen value 
according to a position of the pixel in the set of digital image data; generating a screen 
amplitude weighting value based on the values in the membership vector for the pixel; 
multiplying the screen value and the screen amplitude weighting value to produce a 
modified screen value; and adding the modified screen value to the pixel of image data. 
The filtering process filters the pixel by low-pass filtering the pixel; high-pass filtering 
the pixel; non-filtering the pixel; multiplying each filtered pixel by a gain factor based 
on the values in the membership vector associated with the pixel; and adding the 
products to produce a filtered pixel of image data. 

[0010] The present invention contemplates new and improved 

methods for classifying images that overcome the above-referenced problems and 
others. 
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Summary of the Invention 

[001 1] In accordance with one aspect of the present invention, 

a method for classification of an image is provided. The method is comprised of: a) 
extracting a plurality of features from an input image; and b) classifying the input image 
in picture or graphics classes using a combination of two or more of the extracted 
features. 

[0012] In accordance with another aspect of the present 

invention, a method for evaluating the confidence level of the classification of an image 
is provided. The method is comprised of: a) extracting a plurality of features from an 
input image; b) classifying the input image in picture or graphics classes using at least 
one of the extracted features to; and c) determining the confidence level of the 
classification using a combination of two or more of the extracted features. 

[0013] In accordance with another aspect of the present 

invention, a method for classification of an input image in natural picture or synthetic 
graphics classes is provided. The method is comprised of: a) extracting one or more 
spatial gray-level dependence texture features from the input image; b) processing each 
extracted feature using an algorithm associated with the feature; c) comparing the result 
of each feature algorithm to one or more previously selected thresholds; and d) if, 
according to previously determined rules, any comparison is determinative of the class 
of the input image, classifying the input image in either the natural picture or synthetic 
graphics class according to the previously determined rules, otherwise indicating the 
result is indeterminate. 

[0014] In accordance with another aspect of the present 

invention, another method for classification of an input image in natural picture or 
synthetic graphics classes is provided. The method is comprised of: a) extracting one 
or more color discreteness features from the input image; b) processing each extracted 
feature using an algorithm associated with the feature; c) comparing the result of each 
feature algorithm to one or more previously selected thresholds; and d) if, according to 
previously determined rules, any comparison is determinative of the class of the input 
image, classifying the input image in either the natural picture or synthetic graphics 
classes according to the previously determined rules, otherwise indicating the result is 
indeterminate. 



[0015] In accordance with another aspect of the present 

invention, another method for classification of an input image in a synthetic graphics 
class is provided. The method is comprised of: a) extracting one or more edge features 
from the input image; b) processing each extracted feature using an algorithm 
associated with the feature; c) comparing the result of each feature algorithm to one or 
more previously selected thresholds; and d) if, according to previously determined rules, 
any comparison is determinative of the class of the input image, classifying the input 
image in either the natural picture or synthetic graphics classes according to the 
previously determined rules, otherwise indicating the result is indeterminate. 

[0016] In accordance with another aspect of the present 

invention, another method for classification of an input image in natural picture or 
,JJ synthetic graphics classes is provided. The method is comprised of a) extracting a 

2 plurality of features from an input image; and b) processing two or more extracted 

! 4f features using a neural network to classify the input image in either natural picture or 

jfgi synthetic graphics classes. 

m [0017] In accordance with another aspect of the present 

G invention, an image processing system for producing an output image associated with 

i!V an input image based on classification of the input image is provided. The system is 

q comprised of a feature extractor for extracting a plurality of features from the input 

^ image; a binary classifier for classifying the input image in natural picture or synthetics 

graphics classes using a combination of any two or more of the extracted features; a 
picture processing module for processing the input image using picture image 
processing functions; a graphics processing module for processing the input image using 
graphics image processing functions; and a switch for routing the input image for image 
processing by the picture processing module or the graphics processing module based 
on the classification of the input image by the binary classifier in either natural picture 
and synthetic graphics classes. 

[0018] In accordance with another aspect of the present 

invention, a method for classification of areas of an input image in picture, graphics, or 
fuzzy classes is provided. The method is comprised of: a) extracting a plurality of 
features from an input image; and b) processing two or more extracted features using 
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a soft classifier to classify areas of the input image in either picture, graphics, or fuzzy 
classes. 

[0019] In accordance with another aspect of the present 

invention, an image processing system for producing an output image associated with 
an input image based on classification of areas of the input image is provided. The 
system is comprised of: a feature extractor for extracting a plurality of features from the 
input image; a soft classifier for classifying areas of the input image in picture, graphics, 
or fuzzy classes using a combination of any two or more of the extracted features; a 
plurality of image processing modules for providing a plurality of image processing 
functions; and a blender for blending the image processing functions from the image 
processing modules, said blending based on the classification of areas of the input image 
by the soft classifier. 

[0020] One advantage of the present invention is that an input 

image is classified as either a natural picture or synthetic graphics with less error than 
prior classifiers by using new features for classification. 

[0021] Another advantage of the present invention is that an 

input image is classified as either a natural picture or synthetic graphics with less error 
than prior classifiers by using combinations of features for classification. 

[0022] Another advantage of the present invention is that an 

input image is classified by a "soft" classifier using new features and combinations of 
features to classify areas of the image as either picture, graphics, or fuzzy classes. 

[0023] Another advantage of the present invention is that the 

"soft" classifier is able to predict a confidence level for picture and graphics image 
classification. 

[0024] Another advantage of the present invention is that image 

processing functions are blended in conjunction with picture, graphics, and fuzzy 
classifications of image areas by the "soft" classifier to produce a more desirable output 
image than prior image processing systems. 

[0025] Still further advantages and benefits of the present 

invention will become apparent to those of ordinary skill in the art upon reading and 
understanding the following detailed description of the preferred embodiments. 
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Brief Description of the Drawings 

[0026] The invention may take form in various components and 

arrangements of components, and in various steps and arrangements of steps. The 
drawings are only for purposes of illustrating preferred embodiments and are not to be 
construed as limiting the invention. 

[0027] FIG. 1 is a flowchart of an image classification process 

using SGLD texture features in accordance with an embodiment of the present 
invention; 

[0028] FIG. 2 is a flowchart of the SGLD matrix initialization 

and construction process in accordance with an embodiment of the present invention; 

[0029] FIG. 3 is a flowchart of an image classification process 

% ? using color discreteness features in accordance with an embodiment of the present 

^ invention; 

Ijt [0030] FIG. 4 is a flowchart of an image classification process 

using edge features in accordance with an embodiment of the present invention; 
* u [003 1] FIG. 5 is a flowchart of an image classification process 

O using a combination of SGLD texture features, color discreteness features, and edge 

i| features in accordance with an embodiment of the present invention; 

51 [0032] FIG. 6 is a block diagram of an image processing system 

!M* using a "binary" image classification process (i.e., classification of images between 

picture or graphics classes); and 

[0033] FIG. 7 is a block diagram of an image processing system 

using a "soft" image classification process (i.e., classification of image areas between 
picture, graphics, or fuzzy classes) and an associated process for blending image 
processing functions based on the classification. 



Detailed Description of the Preferred Embodiments 

[0034] Spatial gray-level dependence (SGLD) techniques for 

image analysis are well known. SGLD feature extraction creates a two-dimensional 
histogram that measures first and second-order statistics of an image. These features 
are captured in SGLD matrices. This was originally proposed for texture analysis of 
multi-level images. Additionally, since texture features distinguish natural pictures from 
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synthetic graphics, SGLD techniques can be applied to picture/graphics classification 
of images. A picture/graphics classifier can be created with algorithms that analyze the 
texture features captured in SGLD matrices. Using the SGLD texture features, the 
classifier works to determine whether a scanned image is a natural picture or synthetic 
graphics. Furthermore, in color images, the luminance component typically contains 
enough information to determine the origin of the image. Therefore, an SGLD matrix 
that captures the luminance component of an image and a picture/graphics classifier 
using the luminance component from the matrix in a classification algorithm can 
determine whether the image is a natural picture or synthetic graphics. 

[0035] With reference to FIG. 1, a flowchart of an image 

classification process using SGLD texture features 100 in accordance with an 
embodiment of the present invention is shown. Generally, the classification process 
filters an input image to smooth out halftones, builds an SGLD matrix from the 
smoothed image, extracts texture features from the matrix, and performs an algorithm 
to determine whether the image is a natural picture or synthetic graphics based on one 
(1) or more of the texture features. 

[0036] More specifically, the process 100 begins with an input 

image 102. The image is processed using a low-pass filter 104 (e.g., a W x W 
averaging filter) to smooth the luminance component and reduce any halftone noise. 
The SGLD matrix is basically a GL x GL two-dimensional histogram, where GL is the 
number of gray levels (e.g., 256), The SGLD matrix is generated by first performing 
an initialization (e.g., set to zero) 106. Next, the SGLD matrix is built from the 
smoothed image 108. The SGLD matrix is a two-dimensional histogram corresponding 
to certain characteristics of the pixels in the input image. For each pixel (m, n) in the 
smoothed image, a neighboring value is calculated using the following logic and 
equations: 

if |x(m, n + d) - x(m, n)| > |x(m + d, n) - x(m, n)| 
then y(m, n) = x(m, n + d), 

otherwise y(m, n) = x(m + d, n), (1), 
where x(m, n) is the smoothed pixel value at (m, n), (m, n + d) and (m + d, n) are 
vertical and horizontal neighbors, respectively, and d is a fixed integer (typically 1 or 
2). 
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[0037] With reference to FIG. 2, a flowchart of an embodiment 

of the SGLD matrix initialization and construction process is shown. The initialization 
step 106 sets the SGLD matrix to zero (0) and sets a pixel counter (N) to zero (0) 154. 
The SGLD matrix is constructed from a low-pass filtered image 152 provided by the 
low-pass filter 104. Construction of the SGLD matrix begins by getting a pixel (m, n) 
156 from the filtered image. A neighboring value for the pixel (m, n) is calculated 
using the algorithm in equation (1). If |x(m, n + d) - x(m, n)| > |x(m + d, n) - x(m, n)| 
158, then y(m, n) - x(m, n + d) 160. Otherwise, y(m, n) = x(m + d, n) 162. As is 
apparent, if pixel (m, n) is in a flat area where x(m, n) is equal to y(m, n), the entry 
[x(m, n), y(m, n)] is on the diagonal. On the other hand, if (m, n) is on an edge, the 
difference between x(m, n) and y(m, n) will be significant, and [x(m, n), y(m, n)] will 
be far away from the diagonal. 

[0038] The entry [x(m, n), y(m, n)] in the SGLD matrix is then 

increased by one (1) and the pixel counter (N) is increased by one (1). Next, a check 
is made to determine if the calculation was for the last pixel 166 of the input image. If 
so, SGLD matrix construction is complete and the SGLD matrix is ready for feature 
extraction 168. Otherwise, the next pixel is retrieved 156 from the input image. 

[0039] For the matrix, the neighboring pixels in graphics images 

are expected to be either correlated or very different. In other words, for graphics 
images, SGLD matrix entries are usually either on the diagonal or far away from the 
diagonal. This is because most pixels are either at the flat regions or on the edges. On 
the other hand, pixels of natural pictures are not expected to have many abrupt changes. 
Accordingly, masses are expected to be concentrated at the entries that are near the 
diagonal for picture images. This shows the noisy nature of the picture images. 

[0040] Returning to FIG. 1, many features (e.g., variance, bias, 

skewness, fitness) can be extracted from the SGLD matrix to classify the input image 
between picture and graphics. The features can be implemented individually or 
combined in various methods (e.g., linear combination). Once the SGLD matrix is built, 
a feature or combination of features is selected for extraction 1 10 and processed using 
feature algorithms. For example, a first feature algorithm measures variance (V) (i.e., 
the second-order moment around the diagonal) 1 12 and is defined as: 
V = Z !n . m! > A s(m, n) (m - n) 2 /N (2), 



where s(m, n) is the (m, n)-th entry of the SGLD matrix, A is an integer parameter 
typically between 1 and 16 and; 

N=2| n . m! > A s(m,n) (3). 

[0041] As the summation is over all (m, n) such that |m - n| > A, 

all the pixels in the flat regions are ignored. For graphics images, the remaining pixels 
are on the edges, while for picture images, both pixels in the noisy regions and pixels 
on the edges are included. Variance (V) is typically larger for graphics images than for 
picture images. 

[0042] The second feature algorithm measures average bias (B) 

114 and is defined as: 

B= S, n . m | > A s(m, n) [n-^i(m)] 2 /N (4), 
where |u(m) is the mean of s(m, n) for a fixed m. For a given m, the distribution of s(m, 
n) is roughly symmetrical about the diagonal for picture images, as noise typically has 
a zero mean symmetrical distribution. As a result B is usually small for picture images. 
For graphics images, s(m, n) is usually unsymmetrical and B is large. 

[0043] The third feature algorithm measures skewness (S) 116 

and is defined as: 



S = skewness == ^ 



GL-\ 



- m\(n - m)s(m,ri) 



m=0 



^{ri)IC, (5), 



m=0 

where: 

GL-l GL-1 

c(n) = ]Ts(m, n) and C = 5>(n). (6). 

m=0 n=0 

[0044] The fourth feature algorithm measures fitness (F) 118 

and is defined to be: 

GL-\ 



F = fitness = -=2 ? (7), 



where a is defined such that: 
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- J)] = 0.6xC. 



]T \s(m, m + d) + s(m, m 

d=0 



(8). 



[0045] 



The image type decision 120 compares the result of the 



feature algorithm(s) to previously selected low and high thresholds (i.e., TL and TH, 
respectively) depending on the algorithm(s) and combinations selected. If the result of 
the feature algorithm(s) is below the low threshold (TL), the image is classified as a 
natural picture 122. If the result exceeds the high threshold (TH), the classification is 
synthetic graphics 126. Obviously, if the behavior of a particular feature is converse to 
this logic, the decision logic can be easily reversed to accommodate. If the result of the 
feature algorithm^) is equal to or between the low and high thresholds, the class of the 
image cannot be determined (i.e., indeterminate 124) from the feature or combination 
of features selected. It is understood that a number of other alternatives are possible. 
For example, a result equal to a particular threshold can be said to be determinative of 
the image class, rather than indeterminate. Also, in certain circumstances the low and 
high threshold can be equal 



classification process using color discreteness features 200 in accordance with an 
embodiment of the present invention is shown. The process 200 begins with an input 
image 202. First, the input image is transformed into a color space 204, in which the 
classification is performed. Although CIELUV space is used as one embodiment, many 
other color spaces can also be used. Next, the image is smoothed using an averaging 
filter 206 to remove any noise due to halftones. For example, a 4 x 4 filter was used 
successfully. Color histograms are computed for each of the three (3) color channels 
(i.e., luminance (L), U, and V) 208. The L, U, and V histograms are normalized 210 
by the number of pixels in the image. The color representation scheme is invariant 
under rotation and translation of the input image and the normalization provides scale 
invariance. If (i) is the histogram of an image, where the index i represents a histogram 
bin, then the normalized histogram H is defined as follows: 



[0046] 



With reference to FIG. 3, a flowchart of an image 



(9). 



GL-1 
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[0047] Since graphics are generated using a limited number of 

colors, graphics images usually are comprised of a few areas of uniform color. Hence, 
the color histograms for a graphics image usually contain several sharp peaks. On the 
other hand, natural pictures usually contain more colors with smoothly varying 
transitions. Hence, natural pictures are more noisy and produce histograms containing 
fewer and smoother peaks. This difference in the histograms is captured in color 
discreteness algorithms for each color channel (i.e., R_L algorithm 212, R_U algorithm 
214, and R_V algorithm 216). The color discreteness algorithms are defined as follows: 

GL-l 

R_L= ^\H _L(i + \)-H _L(i)\ (10), 

2=1 

GL-l 

R_U=^\H _U(i + l)-H_U(i) (11), 

GL-l 

R_V=^\H_V(i + i)-H_V(i)\ (12), 

1=1 

where GL is the number of bins in the H_L, H_U, and H_V color histograms (typically, 
256). 

[0048] The image type decision 218 compares the results of the 

color discreteness algorithms to previously selected thresholds (e.g., low threshold (TL) 
and high threshold (TH)). If the result of any color discreteness algorithm is above TH 
or below TL, the image is classified as either a graphics 224 or picture 220 according 
to predetermined rules. Otherwise, the class of the image cannot be determined (i.e., 
indeterminate 222) by color discreteness features. Alternatively, the classifier may use 
all three (3) color discreteness features (as described above), any combination of two 
(2) features, or any one (1) feature. The color discreteness features can be computed 
faster than texture features (discussed above) or edge features (discussed below). 

[0049] With reference to FIG. 4, a flowchart of an image 

classification process using edge features 300 in accordance with an embodiment of the 
present invention is shown. The process 300 begins with an input image 302. First, 
edges of color areas in the image are detected 304 using a standard Canny edge 
detector and an edge map image is created. The parameters identified for the edge 
detector were determined empirically. Deviations that produce suitable results are also 
contemplated. Next, the edges in the edge map image are connected 306 (e.g., using 
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a standard 8-connected component algorithm). The average number of pixels per 

connected edge (E) in the edge map image is used as a feature 308. The algorithm for 

this edge feature is defined as: 

2 No. of Edge Pixels 
No. of Connected Edges 

[0050] Typically, graphics have fewer connected edges, but each 

connected edge consists of a large number of pixels. On the other hand, pictures have 
a lot more connected edges, but usually very few pixels in each connected edge. This 
feature is particularly accurate for high values. In other words, if the value of E is high, 
it is almost certain that the image is graphics. However, if the value of E is low, nothing 
can be said about the image. This is because the E value may be low for graphics that 
have low frequency halftones or certain background. Accordingly, the image type 
decision 310 compares the result of the feature algorithm to a previously selected high 
threshold (i.e., TH). If the result exceeds the high threshold (TH), the classification is 
synthetic graphics 314. Otherwise, the class of the image cannot be determined (i.e., 
indeterminate 312). It is understood that other alternatives are possible. For example, 
horizontal or vertical edges in the edge map may be used to classify images because the 
features are much more predominant in synthetic graphics than in natural pictures. Any 
combination of edge features or any one (1) edge feature can be used by the classifier. 

[0051] With reference to FIG. 5, a flowchart of an image 

classification process using a combination of SGLD texture features, color discreteness 
features, and edge features 400 in accordance with an embodiment of the present 
invention is shown. Notably, this image classifier combines all the features of the three 
(3) classifiers discussed above. SGLD texture, color, or edge features may be combined 
into one (1) classifier, whereby performance may be improved over classifiers using a 
single feature. 

[0052] While developing a classifier based a combination of 

texture, color, and edge features, it was observed that the classification and regression 
tree (CART) method, a public domain tree classifier, gave significant importance to the 
first color discreteness feature (R_L). It was also observed that the edge feature (E) 
was only accurate at large values (i.e., if the feature value was large) in determining that 
the image was a graphics. However, when the edge feature value was small, it was 
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unable to determine whether the image was a picture or a graphics. All these 
observations can be combined in a rule-based tree classifier that uses a neural network 
at one (1) of its nodes. The combination of classifiers can analyze texture, color, and 
edge features to distinguish between picture and graphics images. 

[0053] The process 400 begins with an input image 102. Next, 

the features are extracted from the input image 404. Feature extraction includes 
compiling SGLD texture features 406 (e.g., variance (V), bias (B), skewness (S), fitness 
(F)) ? color discreteness features 408 (e.g., R_L, R_U, R_V), and edge features 410 
(e.g., pixels per connected edge (E), horizontal edges, vertical edges). Alternatively, 
any combination of two (2) or more features that lead to the desired classification are 
contemplated, including the use of additional features. The SGLD texture features are 
compiled by performing steps 104-1 18 of the process depicted in FIG. 1. Similarly, the 
color discreteness features are compiled by performing steps 204-216 of the process 
depicted in FIG. 3. Likewise, the edge features are compiled by performing steps 304- 
310 of FIG. 4. 

[0054] While developing the classifier, it was observed that the 

edge feature (E) was accurate at large values (i.e., when E is large, it is almost certain 
that the image is graphics). This observation was incorporated as a rule in the classifier. 
Hence, a first rule-based decision (i.e., E > TE 412) classifies the image as graphics 
420, if: 

E > TE (14), 

where TE is a previously identified high threshold value for the edge feature. 

Experimentally, TE = 120 produced satisfactory results. 

[0055] It was also observed that the public domain tree classifier 

CART gave significant importance to the first color discreteness feature (R_L). This 
observation was also incorporated as a rule in the classifier. Hence, a second rule-based 
decision (i.e., R_L > TH, R_L < TL 414) classifies the image as a graphics, if: 
R_L > TH (15), 
and as picture, if: 

R L < TL (16), 
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where TH and TL are high and low threshold values, respectively, for the R_L color 
discreteness feature. Experimentally, TH - 0.15 and TL = 0.05 produced satisfactory 
results. 

[0056] If the class of the image cannot be determined from the 

rules the neural network 416 operates using any combination of two or more of the 
texture, color, and edge features to make the determination. The features are scaled to 
[0, 1] before feeding into the neural network. One embodiment of the neural network 
is a standard feedforward architecture. A back-propagation algorithm is implemented 
for training the network. The feedforward architecture includes an input layer, a hidden 
layer, and an output layer. The input layer includes a plurality of source nodes (e.g., 
eight (8)). The hidden layer and the output layer are each comprised of one (1) neuron 
(i.e., computation nodes). The source nodes are projected onto the computation nodes, 
but not vice versa - hence the "feed forward" name. The hidden neuron intervenes 
between the external input and output layers and enables the network to extract higher- 
order statistics, 

[0057] The back-propagation algorithm, also known as the error 

back-propagation algorithm, trains the neural network in a supervised manner. 
Basically, back-propagation learning consists of two (2) passes through the different 
layers of the network: a forward pass and a backward pass. In the forward pass, an 
input pattern is applied to the source nodes and its effect propagates through the 
network. The output produced represents the actual response of the network. During 
the forward pass the synaptic weights of the network are all fixed. During the 
backward pass, on the other hand, the synaptic weights are all adjusted in accordance 
with an error-correction rule. Specifically, the actual response of the network is 
subtracted from a desired (target) response to produce an error signal. This error signal 
is then propagated backward through the network, against the direction of synaptic 
connections - hence the name "error back-propagation." The synaptic weights are 
adjusted to make the actual response of the network move closer to the desired 
response in a statistical sense. 

[0058] As shown in FIG. 5, the neural network has eight (8) 

inputs 404 (i.e., V, B, S, F, R_L, RU, R_V, E) and one (1) binary output (i.e., 
picture/graphics 422). The rule-based portion of the classifier (i.e., 412, 414) does not 
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need any training. The neural network 416 was trained with samples that were already 
classified correctly by the rule-based classifier portion and tested on the rest of the 
samples. 

[0059] With reference to FIG. 6, a block diagram of an image 

segmentation system 500 using a "binary" image classification process (i.e., 
classification of images between picture or graphics classes) is shown. The 
picture/graphics classifiers (i.e., 100, 200, 300, 400) of FIGS. 1-4 are "binary" 
classifiers and could be implemented in such a system 500. As described above for 
FIGS. 1-4, an input image 502 is provided to a feature extractor 504. The feature 
extractor 504 extracts pertinent characteristics (i.e., features) based on the parameters 
required by algorithms of the binary classifier 506. The binary classifier 506 exercises 
algorithms designed to classify the input image between a natural picture or a synthetic 
graphics image (e.g., [0, 1] where 0 indicates picture and 1 indicates graphics). This 
binary classification result is provided to a switch 508. The switch 508 receives the 
input image 502 and switches it between picture processing 510 and graphics 
processing 512, depending on the binary classification result. Picture processing 510 
processes the image in a manner tailored to maximize the quality of natural picture 
images (e.g., gamut mapping). Similarly, graphics processing 512 is tailored to 
maximizes the quality of synthetic graphics images (e.g., filtering). If the input image 
is classified as a picture, the input image 502 is switched to picture processing 510 and 
a picture output 514 is produced. Alternatively, if the image is classified as graphics, 
the input image 502 is switched to graphics processing 512 and a graphics output 516 
is produced. In the event that the binary classifier 506 cannot determine the class of the 
input image, one (1) of the processes (e.g., picture processing 510) may be selected by 
default. 

[0060] With reference to FIG. 7, a block diagram of an image 

processing system using a "soft" image classification process (i.e., classification of 
image areas between picture, graphics, or fuzzy classes) and an associated process for 
blending image processing functions based on the classification is shown. 

[0061] The "soft" fuzzy image classification is an improvement 

over the fuzzy classification process (e.g., as disclosed in U.S. Patent No 5,778,156 to 
Schweid) by making the classification decision "soft." This is done by using a neural 
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network, with image features as inputs and "two" outputs. The soft classification result 
is then used to "blend" the down stream image processing functions (i.e., gamut 
mapping or filtering). It can also be used to evaluate the confidence level of the 
classification, and take appropriate actions. Again, as described above for FIGS. 1-4, 
an image input 602 is provided to a feature extractor 604. The feature extractor 604 
extracts two (2) or more pertinent characteristics (i.e., features) from the input image 
602 and provides it to a soft classifier 606 (e.g., neural network, fuzzy decision tree, 
Gaussian maximum likelihood, or any classifier with continuous, rather than binary 
output). As discussed above for binary classifiers, the features provided to the classifier 
can be indicative of various distinguishing characteristics of an input image. For 
example, two (2) or more texture (e.g., V, B, S, F), color discreteness (e.g., R_L, R_U, 
R_V), or edge (e.g., E) features can be implemented in any combination. Additional 
features that lead to the desired classification are also contemplated. 

[0062] In one embodiment, the soft classifier 606 is a neural 

network in a standard feedforward architecture, similar to the neural network described 
above in reference to FIG. 5. However, in the neural network of the "soft" classifier, 
the hidden layer includes one (1) or two (2) neurons and the output layer is comprised 
of two (2) neurons. Like the neural network above, a back-propagation algorithm is 
implemented for training the network. Each of the two (2) outputs (i.e., a, b) of the 
neural network will have a value that ranges between a minimum and a maximum (e.g., 
between 0 and 1). The output value represents the level of membership for an area of 
the input image in each of two (2) classes (e.g., picture, graphics). Ideally, when an 
area is in the graphics class, the output will be [1, 0]. Conversely, if the area is in the 
picture class, the output will be [0, 1], In actual cases using these rules, both outputs 
(e.g., [a, b]) will usually range between 0 and 1, indicating that the area of the input 
image is in the fuzzy class and further indicating the level of membership to both picture 
and graphics classes. The "soft" classification result 608 (i.e., an input image with 
picture, graphics, and/or fuzzy areas) is used to "blend" 610 the down stream image 
processing functions (e.g., image processing 1 (612), image processing 2 (614)), 
creating a "blended" image processing function, to produce an output image 616. In 
the preferred embodiment, image processing 1 is a gamut mapping/filtering process for 
picture class and image processing 2 is a gamut mapping/filtering process for graphics 

-17- 



class. However, alternative configurations are envisioned with additional image 
processing functions or different functions. 

[0063] In another embodiment, the input image 602 is provided 

to each of a plurality of image processing functions (e.g., image processing 1 (612) and 
image processing 2 (614)), rather than to the "blender." This is shown in FIG. 7 via 
dashed lines. In this alternative, the "soft" classification result 608 (i.e., an input image 
with picture, graphics, and/or fuzzy areas) is used to "blend" 610 the processed images 
resulting from the multiple image processing functions to produce a "blended" output 
image 616. 

[0064] Alternatively, if a binary decision is desired, a and b are 

compared to make the classification decision. The difference between a and b provides 
the classification based on the following rules: 

a-b » 0, graphics class (17); 
a-b = 0, indeterminate (18); and 

a-b « 0, picture class (19). 

[0065] The difference between a and b can also be used as a 

confidence level of the classification based on the following rules: 
a-b » 0, strong confidence of graphics class, little confidence of picture class; (20); 
a-b = 0, uncertainty in classification; (21); and 

a-b « 0, strong likelihood of picture class, little confidence of graphics class (22). 

[0066] In some spatial gamut mapping techniques, a spatial 

feedback filter is used to preserve luminance variations in the gamut mapping process. 
The optimal footprint and coefficients of the filter depend heavily on the nature of the 
image content (i.e., natural picture versus synthetic graphics). Where this technique is 
implemented by blending 601, the output of the soft classifier 608 can be used to steer 
the filter parameters. Similarly, methods of blending filter coefficients have been 
described in U.S Patent 5,778,156 to Schweid et al. entitled "Method and System for 
Implementing Fuzzy Image Processing of Image Data." 

[0067] Even if downstream image processing functions (e.g., 

gamut mapping or filtering) are not blended 610, the "soft" classification result 608 can 
be used to bias the classification decision to be on the safe side or to select a safe or 
neutral position when the confidence level is low. 
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[0068] The invention has been described with reference to the 

preferred embodiments. Obviously, modifications and alterations will occur to others 
upon reading and understanding the preceding detailed description. It is intended that 
the invention be construed as including all such modifications and alterations insofar as 
they come within the scope of the appended claims or the equivalents thereof. 
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